Multimodal Function Tool Results

Problem

FunctionToolResult can represent images in Eino, but the legacy Ark Responses SDK exposes function_call_output.output as a string. Image tool results therefore failed before reaching an Ark vision model.

Solution

Route AgenticArk requests through ark-runtime-go while retaining the existing public configuration and option types. A text-only result remains a string. A result containing an image is sent as an ordered input_text / input_image content array.

Decisions

• Keep the image attached to function_call_output; do not turn it into an unrelated user message.
• Reject file:// media inputs before the new SDK can upload local files implicitly.
• Escape the private bridge marker in plain text and rewrite JSON as raw messages so user text and large integers are preserved exactly.

Key Insight

Tool output modality is part of the conversation protocol. Passing an image as a normal message loses the model's association with the originating tool call; the Responses API must receive it in that call's output item.

Summary

多模态函数工具结果

问题

Eino 的 FunctionToolResult 可以表达图片，但旧 Ark Responses SDK 将 function_call_output.output 定义为字符串，导致图片工具结果无法传递给支持视觉能力的 Ark 模型。

方案

内部请求改由 ark-runtime-go 发送，同时保留现有公开配置和 option 类型。纯文本结果仍发送为字符串；包含图片的结果按顺序发送为 input_text / input_image 内容数组。

决策

• 图片保持在 function_call_output 内，不伪装成无关的 user message。
• 在新 SDK 自动上传前拒绝 file:// 媒体输入，避免隐式上传本地文件。
• 对内部 bridge 标记做文本转义，并以 json.RawMessage 改写请求，保证用户文本和大整数不被破坏。

关键洞察

工具输出的模态属于对话协议。将图片作为普通消息传递会丢失其与原工具调用的关联；Responses API 必须在对应工具调用的输出项中接收图片。

总结